Systems and Methods for Semantically Classifying and Extracting Shots in Video

ABSTRACT

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 U.S.C. §119(e) of U.S.Provisional Patent Application No. 61/029,042, filed Feb. 15, 2008, andentitled “Scene Classification on Video Data with a Material ModelingStep.” This application claims the benefit under 35 U.S.C. §121 of U.S.patent application Ser. No. 12/372,561, filed Feb. 17, 2009, andentitled “Systems and methods for semantically classifying shots invideo.” Each application of which is incorporated herein by reference asif set forth herein in its entirety.

TECHNICAL FIELD

The present systems and methods relate generally to classification ofvideo data, files, or streams, and more particularly to semanticclassification of shots or sequences in videos based on video contentfor purposes of content-based video indexing and retrieval, as well asoptimizing efficiency of further video analysis.

BACKGROUND

Image classification systems (i.e. systems in which the content of asingle image or photograph is analyzed to determine an appropriate labelor descriptor for the image) are known in the art. Such systems aregenerally used to label or classify images according to predefinedtextual descriptors. Typically, an image classification system analyzesan image via the use of one or more “classifier” algorithms (describedin greater detail below) that identify a predefined label that matchesor partially matches an image based on the image content and associatethe identified label with the image. For example, an image of a horse ona farm may be labeled “horse,” or “farm,” or both. In some systems, animage or photo may be labeled according to broad categories of imagecontent (e.g. indoor or outdoor, city or landscape, etc.), whereas othersystems utilize more narrow categories (e.g. desert, ocean, forest, car,person, etc.). Some systems even classify images based on identifiedpersons in the image (e.g. celebrities, political figures, etc.),objects in the image, etc. These labels or classifications are usefulfor a variety of purposes, such as association with metadata tags orother identification mechanisms for use in image indexing and retrievalsystems, surveillance and security systems, and other similar imagerecognition purposes.

Such image classification systems utilize a variety of methods toclassify images, with varying results. One such technique involvesexamining the power spectrum of an image in conjunction with PrincipalComponents Analysis (PCA) to identify the type of content in the image,as described in A. Torralba and A. Oliva, Statistics of Natural ImageCategories, Network: Computation in Neural Systems, vol. 14, pp. 391-412(2003). Other approaches include using a “bag of words” with ScaleInvariant Feature Transform (SIFT) descriptors (see P. Quelhas and J.Odobez, Natural Scene Image Modeling Using Color and Texture Visterms,Conference on Image and Video Retrieval (CIVR), Phoenix Ariz. (2006)) incombination with Latent Dirichlet Allocation (see L. Fei-Fei and P.Perona, A Bayesian Hierarchical Model for Learning Natural SceneCategories, IEEE Conference on Computer Vision and Pattern Recognition(2005)), probabilistic Latent Semantic Analysis (see A. Bosch et al.,Scene Classification Via pLSA, ECCV (4), pp. 517-30 (2006)), or aspatial pyramid (see S. Lazebnik et al., Beyond Bags of Features:Spatial Pyramid Matching for Recognizing Natural Scene Categories, IEEEConference on Computer Vision and Pattern Recognition, vol. 2, pp.2169-78 (2006)).

Additional approaches to image classification include using aTwo-Dimensional (2D) hidden Markov model (see J. Li and J. Z. Wang,Automatic Linguistic Indexing of Pictures by a Statistical ModelApproach, IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 25, no. 10 (2003)), as well as a wavelet coefficientsrepresentation of features with hierarchical Dirichlet process hiddenMarkov trees (see J. J. Kivinen et al., Learning MultiscaleRepresentations of Natural Scenes Using Dirichlet Processes, IEEE11.sup.th International Conference on Computer Vision (2007)). Stillfurther image classification systems divide an image into a rectangulargrid and classify the proportion of “material” (i.e. category ofcontent) in each grid cell (see, e.g., J. Shotton et al., SemanticTexton Forests for Image Categorization and Segmentation, IEEE ComputerVision and Pattern Recognition (2008); J. Vogel and B. Schiele, NaturalScene Retrieval Based on a Semantic Modeling Step, Conference on Imageand Video Retrieval (CIVR) (2004); etc.). In these systems, theoccurrence of each material over the image is computed and the image isclassified based on the resulting material occurrence vector.

Regardless of the specific approach, conventional image classificationsystems are ill-equipped to classify videos or portions of videos.Conventional systems are designed to analyze individual images in whichcare is taken to carefully frame the subject of the image (i.e. thescene) in a clear manner, whereas videos typically include a variety oftypes of images or frames, many of which are blurry or contain occludedportions. Additionally, the features used in single-image classificationsystems are often designed for narrow and particular purposes, and areunable to identify and classify the wide array of content present inmost videos. Further, even if conventional systems were able to classifyimages from a video, these systems include no defined mechanism toaccount for the presence of a multitude of scene types across a video orportion of video (i.e. identification or classification of a singleimage or frame in a video does not necessarily indicate that the entireshot within the video from which the frame was extracted corresponds tothe identified image class). As used herein, a “shot” defines a unit ofaction in a video filmed without interruption and comprising a singlecamera view.

In addition to those mentioned, classification of video, or shots withinvideo, presents further challenges because of the variations and qualityof images present in most videos. In most video sequences, only part ofthe scene is visible in most frames. As used herein, “scene” refers tothe setting or content of the image or video desirous of classification(i.e. the context or environment of a video shot) (e.g. desert,mountainous, sky, ocean, etc.). In many videos, wide-angle shots areinterspersed with close-up shots. During the close-up shots, the camerais typically focused on the subject of interest, often resulting in ablurred background, thus confusing any part of the scene type that isvisible. Most videos also include shots in which either the camera orobjects within the scene are moving, again causing blurring of theimages within the shot.

Additionally, scene content in videos often varies immensely inappearance, resulting in difficulty in identification of such content.For example, images of buildings vary in size, color, shape, materialsfrom which they are made, etc.; trees change appearance depending on theseason (i.e. leaves change color in the fall, branches become bareduring the winter, etc.); snow may be present in any type of outdoorscene; etc. In addition, the subject of a video shot may be filmed fromdifferent angles within the shot, causing the subject to appeardifferently across frames in the shot. Thus, because video oftenrepresents wide varieties of content and subjects, even within aparticular content type, identification of that content is exceedinglydifficult.

Further, use of raw or basic features, which are sufficient for someconventional image classification systems, are insufficient for a videoclassification system because videos typically include a multiplicity ofimage types. For example, the color distribution may be the same for abeach shot with white sand as for a snow-covered prairie, or an oceanshot compared to a sky shot, etc. Additionally, the mere detection oridentification of a color or type of material in a scene does notnecessarily enable classification of the scene. For example, asnow-tipped mountain covered with forest has a similar distribution ofmaterials and colors as a close-up view of evergreen trees emerging froma snow-blanketed base. Accordingly, the use of strong features, as wellas the spatial arrangement of materials identified by those features, ishelpful in labeling the wide variety of images in video to enableaccurate classification of shots within video.

One system that attempts to overcome the previously-described hurdles inorder to classify videos is the “Vicar” system, described in M. Israelet al., Automating the Construction of Scene Classifiers forContent-Based Video Retrieval, MDM/KDD'04 (2004). The Vicar systemselects one or more representative or “key” frames from a video, anddivides each of the key frames into a grid. Each grid cell is furtherdivided into rectangular “patches,” and each patch is classified into ageneral category (e.g. sky, grass, tree, sand, building, etc.) usingcolor and texture features and a k-Nearest Neighbor classifier. Thefrequency of occurrence of each category in each grid cell is computedand used to classify the overall image. This system infers that if arepresentative frame or frames comprise a certain type of image, thenthe entire shot or video likely corresponds to the same type, and isthus labeled accordingly.

The Vicar system, however, has many drawbacks that produce inconsistentresults. For example, selection of key frames is a relatively arbitraryprocess, and an easily-classifiable frame (i.e. clear, non-occluded,etc.) is not necessarily representative of the scene type(s) associatedwith a shot or video from which the frame was selected. Further, the keyframes are partitioned based on a predetermined grid, such thatresulting grid cells may (and often do) contain more than one category,thus leading to confusion of scene types. Also, the color and texturefeatures used in the system are relatively weak features, which areinadequate for classifying many categories of images. Additionally, theinference that a key frame or frames adequately and accuratelyrepresents an entire sequence of frames does not take into accountvariations in shots, especially for long or extended shots in videos.

Video classification has many practical uses. For example, accurate andefficient classification of video, or shots within video, enablescontent-based video indexing and retrieval. Such indexing and retrievalis useful for cataloguing and searching large databases of videos andvideo clips for use in promotional advertisements, movie and televisiontrailers, newscasts, etc. Additionally, by classifying videos and thusnarrowing the scope of videos that contain certain subject matter,processing times and accuracy of other, related image or video analysisalgorithms is improved. Further, identification and classification ofdisparate shots within a video enables shot boundary detection andindexing associated with the video.

For these and many other reasons, there is a continuing need for asystem or method that accurately and efficiently classifies shots invideo based on a plurality of images or frames associated with the shot.There is a further need for a system that is able to classify shots asbelonging to multiple classes of scene types, and identify particulartimecodes within the video shot at which scene classes vary.

BRIEF SUMMARY OF THE DISCLOSURE

Briefly described, and according to one embodiment, the presentdisclosure is directed to a system for classifying videos based on videocontent, comprising a processor, one or more software modules includingone or more classifiers, and a computer program product. The computerprogram product includes a computer-readable medium that is usable bythe processor, the medium having stored thereon a sequence ofinstructions associated with the one or more software modules that whenexecuted by the processor causes the execution of the steps of receivinga video file, the video file including a plurality of frames; extractinga subset of frames from the video file; if one or more frames in theextracted subset of frames comprises a dark frame, discarding the one ormore dark frames from the subset; and determining whether each frame inthe extracted subset includes content associated with a general contentcategory. The processor causes further execution of the steps of, foreach frame in the extracted subset that includes content associated withthe general content category, generating a scene classification scorevector for the frame via one or more scene classifiers, the sceneclassification score vector including one or more scene classificationscores associated with one or more predefined scene categories withinthe general content category; determining a representative sceneclassification score vector for the video file based on the generatedscene classification score vectors for each extracted frame in thesubset that includes content associated with the general contentcategory; and associating the video file with the one or more predefinedscene categories based on the representative scene classification scorevector.

According to one aspect of the present system, the one or more softwaremodules are selected from the group comprising: an intensityclassification module, an indoor/outdoor classification module, anoutdoor classification module, a segmentation module, a materialarrangement module, and a video file classification module.

According to another aspect, the step of generating the sceneclassification score vector for each frame in the extracted subset offrames that includes content associated with the general contentcategory further comprises the steps of dividing the frame into one ormore segments based on image content in each segment; generating amaterial classification score vector for each segment, each materialclassification score vector including one or more material valuesassociated with one or more predefined material content types, whereineach material value represents a probability that the respective segmentincludes that type of material content; assigning the materialclassification score vector for each segment to each respective pixel inthe segment; generating a material arrangement vector for the framebased on the material classification score vectors assigned to eachpixel; and classifying the material arrangement vector via the one ormore scene classifiers to generate the scene classification score vectorfor the frame.

According to a further aspect, the adjacent segments are combined basedon similar content properties of the segments. In one aspect, the one ormore predefined material content types are selected from the groupcomprising: building, grass, person, road/sidewalk, rock,sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water,and miscellaneous. According to another aspect, the material arrangementvector represents the spatial arrangement of material content in theframe.

According to an additional aspect, a dark frame comprises a frame shotin low or no light. According to another aspect, the processor causesexecution of the further step of determining whether any of the framesin the extracted subset of frames comprises a dark frame. In one aspect,the step of determining whether any of the frames in the extractedsubset of frames comprises a dark frame further comprises the steps of,for each frame, dividing the frame into a plurality of grid cells;calculating an intensity value for each pixel in each cell; calculatingan average intensity value across all pixels in each cell; concatenatingthe average intensity values for each cell in the frame to form anintensity feature vector for the frame; and classifying the intensityfeature vector via an intensity classifier to determine if the framecomprises a dark frame.

According to yet another aspect, the step of determining whether eachframe in the extracted subset includes content associated with thegeneral content category further comprises the steps of extracting aplurality of features from each frame; generating a feature vector foreach frame, wherein each feature vector includes the extracted featuresfor the respective frame; and classifying the feature vector for eachframe via a general category classifier to determine whether each frameincludes content associated with the general content category.

According to still another aspect, the general content categorycomprises outdoor content. In one aspect, the one or more predefinedscene categories are selected from the group comprising: coast/beach,desert, forest, grassland, highway, indoor, lake/river, mountainous,open water, outdoor, sky, snow, and urban.

According to a further aspect, each of the one or more sceneclassification scores represents the probability that a frame includescontent associated with each of the respective predefined scenecategories. In one aspect, each of the one or more scene classificationscores comprises a value greater than or equal to zero and less than orequal to one.

According to an additional aspect, the representative sceneclassification score vector comprises a statistical property of thegenerated scene classification score vectors. In one aspect, thestatistical property is selected from the group comprising: average,median, maximum, and minimum.

According to a yet further aspect, the step of associating the videofile with the one or more scene categories further comprises the stepsof identifying representative scene classification scores in therepresentative scene classification score vector that exceed apredetermined threshold value; and, for each representative sceneclassification score that exceeds the threshold value, associating thevideo file with the one or more predefined scene categories associatedwith the classification scores that exceeded the threshold value.

According to one aspect, the video file comprises a shot of video.

According to a still another aspect, the processor causes furtherexecution of the step of generating a report including the one or morepredefined scene categories associated with the video file.

According to an additional aspect, the one or more scene categoriesassociated with the video file are used for indexing and retrieval ofthe video file.

According to another embodiment, the present disclosure is directed to amethod for classifying videos based on video content, comprising thesteps of receiving a video file, the video file including a plurality offrames; extracting a subset of frames from the video file; if one ormore frames in the extracted subset of frames comprises a dark frame,discarding the one or more dark frames from the subset; and determiningwhether each frame in the extracted subset includes content associatedwith a general content category. The method further comprises the stepsof, for each frame in the extracted subset that includes contentassociated with the general content category, generating a sceneclassification score vector for the frame via one or more sceneclassifiers, the scene classification score vector including one or morescene classification scores associated with one or more predefined scenecategories within the general content category; determining arepresentative scene classification score vector for the video filebased on the generated scene classification score vectors for eachextracted frame in the subset that includes content associated with thegeneral content category; and labeling the video file according to theone or more predefined scene categories based on the representativescene classification score vector.

According to one aspect of the present method, the step of generatingthe scene classification score vector for each frame in the extractedsubset of frames that includes content associated with the generalcontent category further comprises the steps of dividing the frame intoone or more segments based on image content in each segment; generatinga material classification score vector for each segment, each materialclassification score vector including one or more material valuesassociated with one or more predefined material content types, whereineach material value represents a probability that the respective segmentincludes that type of material content; assigning the materialclassification score vector for each segment to each respective pixel inthe segment; generating a material arrangement vector for the framebased on the material classification score vectors assigned to eachpixel; and classifying the material arrangement vector via the one ormore scene classifiers to generate the scene classification score vectorfor the frame.

According to a further aspect, the adjacent segments are combined basedon similar content properties of the segments. In one aspect, the one ormore predefined material content types are selected from the groupcomprising: building, grass, person, road/sidewalk, rock,sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water,and miscellaneous. According to another aspect, the material arrangementvector represents the spatial arrangement of material content in theframe.

According to an additional aspect, a dark frame comprises a frame shotin low or no light. According to another aspect, the method comprisesthe further step of determining whether any of the frames in theextracted subset of frames comprises a dark frame. In one aspect, thestep of determining whether any of the frames in the extracted subset offrames comprises a dark frame further comprises the steps of, for eachframe, dividing the frame into a plurality of grid cells; calculating anintensity value for each pixel in each cell; calculating an averageintensity value across all pixels in each cell; concatenating theaverage intensity values for each cell in the frame to form an intensityfeature vector for the frame; and classifying the intensity featurevector via an intensity classifier to determine if the frame comprises adark frame.

According to yet another aspect, the step of determining whether eachframe in the extracted subset includes content associated with thegeneral content category further comprises the steps of extracting aplurality of features from each frame; generating a feature vector foreach frame, wherein each feature vector includes the extracted featuresfor the respective frame; and classifying the feature vector for eachframe via a general category classifier to determine whether each frameincludes content associated with the general content category.

According to still another aspect, the general content categorycomprises outdoor content. In one aspect, the one or more predefinedscene categories are selected from the group comprising: coast/beach,desert, forest, grassland, highway, indoor, lake/river, mountainous,open water, outdoor, sky, snow, and urban.

According to a further aspect, each of the one or more sceneclassification scores represents the probability that a frame includescontent associated with each of the respective predefined scenecategories. In one aspect, each of the one or more scene classificationscores comprises a value greater than or equal to zero and less than orequal to one.

According to an additional aspect, the representative sceneclassification score vector comprises a statistical property of thegenerated scene classification score vectors. In one aspect, thestatistical property is selected from the group comprising: average,median, maximum, and minimum.

According to a yet further aspect, the step of labeling the video filefurther comprises the steps of identifying representative sceneclassification scores in the representative scene classification scorevector that exceed a predetermined threshold value; and, for eachrepresentative scene classification score that exceeds the thresholdvalue, associating the video file with the one or more predefined scenecategories associated with the classification scores that exceeded thethreshold value.

According to one aspect, the video file comprises a shot of video.

According to still another aspect, the method further comprises the stepof generating a report based on the labeled video file.

According to an additional aspect, the labeled one or more scenecategories are used for indexing and retrieval of the video file.

According to a further embodiment, the present disclosure is directed toa method for classifying videos based on video content, comprising thesteps of receiving a video file, the video file including a plurality offrames, wherein each frame includes a plurality of pixels; andextracting a set of frames from the video file. The method furthercomprises the steps of, for each frame in the extracted set of frames,determining whether the frame comprises a poor classification frame; ifone or more frames in the extracted set of frames comprises a poorclassification frame, removing the one or more poor classificationframes from the set; dividing each frame in the set of frames into oneor more segments, wherein each segment includes relatively uniform imagecontent; and extracting image features from each segment to form afeature vector associated with each segment. The method comprises theadditional steps of generating a material classification score vectorfor each segment via one or more material classifiers based on thefeature vector associated with each segment, wherein each materialclassification score vector includes one or more material classificationscores associated with one or more predefined material contentcategories; and assigning each material classification score vectorassociated with its respective segment to each pixel in each respectivesegment for each respective frame in the set of frames.

According to one aspect, the method further comprises the step ofstoring the material classification score vectors assigned to each pixelin a database for subsequent use in video file classification.

According to another aspect, the method further comprises the step ofcombining adjacent segments based on similar image content featuresextracted from the segments.

According to an additional aspect, poor classification frames aredetermined via one or more classifiers. In one aspect, a poorclassification frame comprises a frame associated with at least one ofthe following frame types: a frame shot in low light, a frame shot atnight, a blurry frame, and an undetermined frame.

According to a further aspect, image features comprise one or morefeatures selected from the group comprising: color features, edgefeatures, line features, texture features, and shape features. In oneaspect, image features comprise data associated with image content.

According to yet another aspect, the one or more material classifiersare hierarchically related.

According to still another aspect, the one or more predefined materialcontent categories are selected from the group comprising: building,grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds,snow/ice, trees/plants, vehicle, water, and miscellaneous.

According to an additional aspect, each of the one or more materialclassification scores represents the probability that a frame includescontent associated with each of the respective predefined materialcontent categories.

According to one aspect, the video file comprises a shot of video.

According to an additional embodiment, the present disclosure isdirected to a method for classifying a video file according to one ormore scene classes, the video file including a plurality of frames,wherein each frame includes a plurality of pixels, and wherein eachpixel is associated with a vector of material classification scoresdescribing the material content in its respective frame. The methodcomprises the steps of: (a) dividing each frame into a plurality of gridcells; (b) for each frame, retrieving the vector of materialclassification scores for each pixel in each cell; (c) for each gridcell, averaging the material classification scores across each pixel inthe cell to form a material occurrence vector for the cell; (d)concatenating the material occurrence vectors for the plurality of gridcells in each frame to generate a material arrangement vector for eachframe; (e) generating a scene classification score associated with eachof the one or more scene classes for each frame in the video file viaone or more scene classifiers based on the material arrangement vectorsgenerated for each frame; (f) generating a representative sceneclassification score for the video file for each of the one or morescene classes based on the scene classification scores generated foreach frame; and (g) if one or more of the representative sceneclassification scores is above a predetermined threshold value, labelingthe video file according to the respective scene classes associated withthe one or more scene classification scores that are above thepredetermined threshold value.

According to one aspect, the method further comprises the step ofrepeating steps (a)-(e) one or more times using varying numbers of gridcells. In one aspect, the one or more scene classifiers comprise aspatial pyramid of classifiers, and wherein varying weights areassociated with the scene classification scores.

According to another aspect, the material content is selected from thegroup comprising: building, grass, person, road/sidewalk, rock,sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water,and miscellaneous.

According to a further aspect, the material occurrence vector representsthe proportion of each type of material content included in the cell. Inone aspect, the material occurrence vectors are concatenated in anordered manner.

According to an additional aspect, each of the material classificationscores represents the probability that a frame includes contentassociated with each of the respective types of material content. In oneaspect, each of the one or more scene classification scores representsthe probability that a frame includes content associated with each ofthe respective one or more scene classes.

According to yet another aspect, the material arrangement vectorgenerated for each frame represents the spatial arrangement of materialcontent in the frame.

According to still another aspect, the one or more scene classes areselected from the group comprising: coast/beach, desert, forest,grassland, highway, indoor, lake/river, mountainous, open water,outdoor, sky, snow, and urban.

According to another aspect, the representative scene classificationscore for each of the one or more scene classes comprises a statisticalproperty of the generated scene classification scores. In one aspect,the statistical property is selected from the group comprising: average,median, maximum, and minimum.

According to a yet further aspect, the video file comprises a shot ofvideo.

According to an additional aspect, the method further comprises the stepof generating a report based on the labeled video file.

According to one aspect, the labeled one or more scene classes are usedfor indexing and retrieval of the video file.

According to still another embodiment, the present disclosure isdirected to a method for labeling videos based on video content,comprising the steps of receiving a video file, wherein the video fileincludes a plurality of frames; extracting a set of frames from theplurality of frames in the video file; for each frame in the extractedset of frames, calculating a probability that the frame includes contentassociated with a predefined scene category; determining arepresentative probability for the set of frames based on the calculatedprobabilities for each frame; and if the representative probabilityexceeds a predetermined threshold, associating the scene category withthe video file.

According to one aspect of the present method, the representativeprobability comprises a statistical property of the calculatedprobabilities for each frame. In one aspect, the statistical property isselected from the group comprising: average, median, maximum, andminimum.

According to another aspect, the method further comprises the step ofindexing the video file according to the associated scene category forsearch and retrieval purposes.

According to an additional aspect, the scene category is selected fromthe list comprising: coast/beach, desert, forest, grassland, highway,indoor, lake/river, mountainous, open water, outdoor, sky, snow, andurban.

According to a further aspect, the calculated probability for each frameis calculated via a classifier.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate one or more embodiments of thedisclosure and, together with the written description, serve to explainthe principles of the disclosure. Wherever possible, the same referencenumbers are used throughout the drawings to refer to the same or likeelements of an embodiment, and wherein:

FIG. 1 illustrates a video classification system according to anembodiment of the present system.

FIG. 2 is a flowchart illustrating the overall functions and processesperformed, from a high-level perspective, by one embodiment of thepresent video classification system.

FIG. 3 illustrates an exemplary frame extracted from a video fileshowing a mountain scene.

FIG. 4 is a flowchart illustrating one embodiment of the intensityclassification process for identifying “dark” frames in a video file.

FIG. 5 is a flowchart illustrating an embodiment of the indoor/outdoorclassification process for classifying frames in a video file as“indoor,” “outdoor,” or “undetermined.”

FIG. 6 is a flowchart illustrating the steps and functions involved inthe outdoor frame classification process according to one embodiment ofthe present system.

FIG. 7 is a flowchart illustrating an embodiment of the segmentcombination/merging process for combining like segments.

FIG. 8 shows an exemplary hierarchical tree representing an organizationof classifiers used to classify image materials.

FIG. 9 is a flowchart illustrating the steps associated with anembodiment of the material arrangement vector generation process.

FIG. 10A is a flowchart illustrating the steps involved according to oneembodiment of the video file classification process for a predefinedshot.

FIG. 10B is a flowchart illustrating the steps involved according to oneembodiment of the video file classification process for shot detection.

FIG. 11 shows a representation of the system components according to oneembodiment of the video classification system.

FIG. 12 shows a confusion matrix of experimental material classificationresults for correctly classified and misclassified images according toone, tested embodiment of the present system.

FIG. 13 is a precision-recall curve of experimental scene classificationresults illustrating precision and recall values for tested images forvarious thresholds according to one, tested embodiment of the presentsystem.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

For the purpose of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings and specific language will be used todescribe the same. It will, nevertheless, be understood that nolimitation of the scope of the disclosure is thereby intended; anyalterations and further modifications of the described or illustratedembodiments, and any further applications of the principles of thedisclosure as illustrated therein are contemplated as would normallyoccur to one skilled in the art to which the disclosure relates.

Overview

Aspects of the present disclosure generally relate to systems andmethods for semantically classifying shots of video based on videocontent. Generally, embodiments of the present system analyze videofiles and associate predefined textual descriptors to the video files.The textual descriptors relate to predefined scene classes or categoriesdescribing content in the files, such as mountain, coast, indoor, urban,forest, and the like. Typically, a video file comprises a shot of video(as defined previously), or a sequence of frames from a video, or anentire video itself. Once classified, the video file may be used for avariety of purposes, including content-based indexing and retrieval,shot boundary detection and identification, and other similar purposes.

Referring now to FIG. 1, an embodiment of the video classificationsystem 10 is shown. The system 10 is shown in an exemplary environmentin which the system interacts with a classified video content user 12.The classified video content user 12 is an entity that has a use forclassified video content, such as a movie or television productionstudio, an advertising agency, and Internet web service contentprovider, or other similar entity. As will be understood and appreciatedby one of ordinary skill in the art, and as shown in FIG. 1, embodimentsof the video classification system 10 comprise computer systemsincluding databases 14 and other storage apparatuses, servers 16, andother components (not specifically shown), such as processors, terminalsand displays, computer-readable media, algorithms, modules, and othercomputer-related components. The computer systems are especiallyconfigured and adapted to perform the functions and processes of thepresent system as described herein (i.e. they are particular machines).Further, any results or outputs 26 relating to classification of videofiles may be stored in a database 16, 18, output as an electronic orprinted report, displayed on a computer terminal, or otherwise deliveredto a system operator or user for analysis, review, and/or furtherprocessing.

As shown in the embodiment of FIG. 1, the classified video content user12 transmits a video file 22 to the video classification system 10 foranalysis and classification. Typically, the video file includes aplurality of frames 24 comprising a plurality of images that togethermake up the video file. As will be understood, the video file 22,resulting output 26, and any other data or files are transmitted betweenthe classified video content user 12 and the video classification system10 via a service oriented architecture (SOA), or some other similar filetransfer protocol. As shown, the exemplary video file includes, at leastin part, images corresponding to a mountain landscape scene. Thisexemplary mountain landscape scene is referenced here and in other partsof this disclosure for illustrative purposes only, and is in no wayintended to limit the scope of the present system.

Upon receipt of a video file 22, the classification system 10 processesthe file (as described in detail below) to identify and classify thefile or shots within the file according to zero or more predefined scenecategories. In some circumstances, based on the nature of the content ofthe video file, no predefined scene category applies to the video. Inother circumstances, multiple classes apply to the given video file.Examples of scene categories include coast/beach, desert, forest,grassland, highway, indoor, lake/river, mountainous, open water,outdoor, sky, snow, urban, and other similar categories as will occur toone of ordinary skill in the art. As will be appreciated, however,embodiments of the present system are not limited to the specific scenecategories mentioned, and other categories are possible according tovarious embodiments and aspects of the present system.

Once processed, the video classification system 10 generates an output26 corresponding to the particular video file 22. Representative outputs26 a, 26 b are presented for exemplary purposes. Output 26 a comprises adata table listing the resulting classification score for each scenecategory for a given video shot. As shown, the table 26 a includes twodata categories or fields: scene class 30 and classification score 32.As will be understood, however, the data categories or files are notlimited to the fields shown, and other embodiments include additionalfields as will occur to one of ordinary skill in the art. As will alsobe understood, although a representative listing of scene classes isshown, actual data tables constructed in accordance with embodiments ofthe present system may include other scene classes not specificallymentioned herein.

According to one embodiment of output 26 a, the classification score 32is a value between 0 and 1 indicating the probability that a particularshot includes content associated with a predefined scene class 30. Aswill be understood, the classification score is represented in a varietyof ways according to various embodiments, such as a percentage, a ratio(as compared to the other scene categories), and other similar ways. Asshown, exemplary table 26 a indicates a hypothetical set ofclassification scores for the mountain shot associated with video file22 and shown in frames 24. The classification scores indicate a highprobability that the scene includes content associated with (andtherefore classified by) mountains (i.e. “mountainous”), “sky,” and a“lake/river” (shown by classification scores 0.91, 0.78, and 0.73,respectively). These scores are as expected, considering the exemplaryimages 24 include mountains, sky, and a lake. Scene category “snow”received a significant score as well (i.e. 0.41), indicating that theshot contains some portion of this type of content.

Once the classification scores are calculated, a threshold value isapplied to the scores to identify the scene classes that likely apply tothe given shot. For example, a system operator may define a thresholdvalue of 0.4, and thus any scene category receiving a classificationscore above the threshold is associated with the shot. Thus, if 0.4 wereused as a threshold, then the shot would be associated with categories“mountainous,” “sky,” “lake/river,” and “snow.” If a higher thresholdwere used, say 0.7, then the shot would be classified as “mountainous,”“sky,” and “lake/river”. A higher threshold might be used, for example,if a system operator desires to label shots only according to contentthat is prominent in the shots. According to one embodiment, thethreshold is varied on a per-class basis. As will be appreciated, thethreshold can be varied at a system operator's discretion to producemore accurate or focused results, include more or fewer classes pershot, etc.

As shown in FIG. 1, output 26 b comprises a data table indicating theidentified class(es) for each shot of video contained in the video file(assuming the video file includes more than one shot), as well as theidentified start and end timecodes for each shot. As shown, the table 26b includes four data categories or fields: shot number 34, starttimecode 36, end timecode 38, and scene class(es) 40. As will beunderstood, however, the data categories or files are not limited to thefields shown, and other embodiments include additional fields as willoccur to one of ordinary skill in the art.

According to various embodiments, table 26 b is used as a subsequentoutput in conjunction with table 26 a after the values in 26 a have beenthresholded for many shots. Or, output 26 b comprises an outputassociated with a shot boundary detection embodiment, in which a videofile 22 comprises many undetected shots, and these shots are identifiedby the video classification system 10 based on variations in sceneclasses. As shown in table 26 b, for example, the system 10 classifiedthe frames associated with hypothetical shot 1 as including “sky”content until 13.12 seconds into the video. At the 13.13 second mark,the classification system 10 identified and classified the shot framesas pertaining to “sky,” “snow,” and “forest.” Thus, the systemdetermined that, based on the change in scene classes, a shot boundaryhad occurred (again, based on some predefined classification scorethreshold value). As will be understood and appreciated, the exemplaryoutputs 26 a, 26 b are presented for illustrative purposes only, andother outputs are possible according to various embodiments of thepresent system.

As shown in FIG. 1, once a video file has been classified by the videoclassification system 10 and a corresponding output 26 or outputs havebeen generated, the classification results are transmitted to theclassified video content user 12 for further use. For example, oneapplication of embodiments of the present system is video indexing andretrieval. In order to accomplish such indexing and retrieval, in oneembodiment, shots and/or videos that have been classified and labeledaccording to predefined scene classes are associated with correspondingmetadata identifiers linked to identified scene classes. These metadataidentifiers are generally stored in index files (e.g. in database 18)and are configured to be searched in order to locate and retrieve videosor shots of videos with the associated classified content. Accordingly,vast amounts of video files 22 (i.e. videos, shots within videos, andsequences of video frames) may be indexed and searched according tocontent in the video files based on the semantic classes linked to themetadata identifiers.

Although the classified video content user 12 is illustrated in theembodiment of FIG. 1 as an entity separate and distinct from the videoclassification system 10, embodiments of the present invention are notlimited to operation with third party entities 12. For example,according to one embodiment, video files 22 and outputs 26 are storedwithin the video classification system database 14, and no interactionwith outside entities is required. Thus, in one embodiment, systemfunctions and processes described herein are carried out entirely withinvideo classification system 10.

For purposes of example throughout this document, exemplary categoriesof scene classes and material classes are given, such as indoor,outdoor, urban, mountainous, highway, vehicle, forest, etc.Additionally, the exemplary embodiment described herein is primarilycouched in terms of a classification system that identifies specificcategories of “outdoor” scenes. It should be understood, however, thatthe present systems and methods are in no way limited to outdoor scenes,and the present systems and methods may be applied to indoor scenes orother types of scenes based on variations in training data, imagefeatures, etc. Accordingly, outdoor video classification systems areoften described herein for illustrative purposes only, but are in no wayintended to limit the scope of the present systems.

FIG. 2 is a flowchart illustrating the overall functions and processes200 performed, from a high-level perspective, by one embodiment of thepresent video classification system 10. The overall process 200 isdescribed initially in a broad sense in conjunction with FIG. 2, and thedetails and specific aspects of each component of the system aredescribed in greater detail below. Starting at step 205, the system 10receives a video file 22 to be processed. As described previously, thevideo file comprises either an entire video, or one or more video shots,or merely a sequence of frames/images. If only a single shot isreceived, then the system classifies the shot as described below. In oneembodiment, a plurality of shots are received (or a file correspondingto an entire video), as well as a list of shot boundaries identifyingthe beginning and ending timecodes for the shots, and each individualshot is classified. According to one embodiment, the shot boundaries inthe list are identified via the algorithm described in Z. Rasheed and M.Shah, Scene Detection in Hollywood Movies and TV Shows, IEEE Conferenceon Computer Vision and Pattern Recognition, vol. 2, pp. 11-343-8 (2003),which is incorporated by reference herein as if set forth herein in itsentirety. As will be appreciated, however, shot boundaries areidentified via other similar mechanisms in other embodiments. In still afurther embodiment, an entire video or sequence of frames is receivedwith no shot boundaries, and the present system 10 identifies shotboundaries based on scene classes changes across the video or framesequence (described in greater detail below).

Regardless of the type of video file received, the system 10 extracts aninitial frame from the video file for processing (step 210). Embodimentsof the present system analyze and classify single frames, and thencombine the results for each analyzed frame to produce an overallclassification or classifications for the shot (described below).Preferably, to reduce overall processing time and increase efficiency,the system only extracts and analyzes a subset of frames in the video,such as one frame from the video file for every ⅓ second of recordedtime. Typically, videos are recorded at a rate of 24 frames/second (or,8 frames per ⅓ second). Thus, a preferred embodiment only analyzes 1 outof 8 frames in a recorded video file. For most applications, a samplingrate of one frame for every ⅓ second of recording time producessatisfactory results, and significantly reduces overall computationtime. As will be understood by one of ordinary skill in the art,however, other sampling rates are possible. In fact, each frame in avideo file 22 may be analyzed if so desired by a system operator.

After a frame has been extracted, the frame is analyzed by an intensityclassification process 400 to determine if the frame is a good candidatefor overall classification. “Dark” frames (i.e. those shot in poorlighting or at night, etc.) are difficult to classify, and thus tend toproduce inconsistent results. Accordingly, if the intensityclassification process 400 determines that a frame is too dark forprocessing (step 215), then the frame is discarded (i.e. not analyzedfurther) (step 220), and a new frame is selected for processing. If,however, the frame is not a dark frame, then the frame is passed throughthe indoor/outdoor classification process 500 to determine whether theframe includes content associated with an indoor scene or outdoor scene.If the frame is not an outdoor frame (as determined by theindoor/outdoor classification process), then the frame is labeled (i.e.classified) as indoor or undetermined, assigned a classification scoreof “0” for all outdoor categories or scene classes (discussed below),and stored in a database 14 (steps 225, 230, 235).

If, however, the frame is in fact an outdoor frame, then the frame isanalyzed by the outdoor classification process 600 to determine whichcategory or categories of material classes apply to the frame. As usedherein, “material” refers to the type or category of content shown in aframe (e.g. sand, grass, rock, building, vehicle, etc.). For example,FIG. 3 illustrates an exemplary frame 24 extracted from a video file 22showing a mountain scene. As shown, the frame includes a variety ofmaterials, such as rock 305, water 310, snow/ice 315, sky/clouds 320,etc. The identified material classes (i.e. classification scores), andtheir spatial arrangement within the frame, are used by subsequentprocesses to classify the entire frame, and eventually the entire shot(described below). Once the material class(es) for the outdoor frame areidentified (as determined by the outdoor classification process), thenthe resulting classification scores are stored in a database 14 (step235 in FIG. 2) for further processing.

Still referring to FIG. 2, at step 240, the system 10 determines whetherany frames are remaining in the video file. If any frames are remaining,then the system extracts the next frame (typically, according to apredefined sampling rate, discussed above), and repeats the steps ofoverall process 200 for the next frame. As will be understood, process200 operates on a looping basis until all selected frames in the videofile have been processed. Once all frames have been analyzed, the videofile classification process 1000, 1001 analyzes the storedclassification scores to classify the video file or shot(s) within thevideo file, and generates a classification output 26 (step 245). After aclassification output has been generated, the process 200 ends.

Feature Extraction

Within embodiments of the present system, “features” are used toidentify content in images/frames, train classifiers to recognize suchimage content, etc. As will be understood and appreciated by those ofordinary skill in the art, a “feature” refers to an individual,measurable heuristic property of an image used in pattern recognitionand classification of the image. Essentially, features are dataextracted from an image region and used to characterize its appearance.

Various types of features are used in image classification systems, suchas color, texture, etc. Features vary in complexity and accuracy (i.e.strong v. weak), producing varying results. Typically, “weak” features,such as raw pixel values, average RGB values in an image region, edgestrength associated with individual pixels, etc., require lesscomputation, but are less accurate as compared to strong features.“Strong” features, such as texture, shape, etc., are typically moredescriptive and better characterize the appearance of an image (i.e. aremore accurate), but usually require more computation and are moredifficult to develop. Preferably, embodiments of the present system usestrong features, but other features are used in various embodiments aswill occur to one of ordinary skill in the art. The preferred embodimentof the present system uses strong color, edge, line, texture, and shapefeatures, as described in further detail below.

Color

According to a preferred embodiment, the color features comprise ahistogram in CIELAB colorspace. As will be understood, a traditional“Lab” colorspace is a color-opponent space with dimension L forbrightness and a and b for the color-opponent dimensions, based onnonlinearly-compressed CIE XYZ color space coordinates. The CIELABcolorspace actually uses the L*, a*, and b* coordinates (as opposed toL, a, and b). Preferably, a three-dimensional (3D) color histogram isformed from the 3-channel color for each pixel in an image using 4 binsfor each channel, resulting in a 64-dimensional histogram. As will beunderstood, while the CIELAB colorspace is preferred, other similarcolorspaces are used for color features according to various embodimentsof the present system.

Edges

According to one embodiment, the edge features comprise edge strengthand edge direction histograms. Preferably, edge strength in each of thex and y directions is computed using the Sobel transform. The computededge strengths are used to form an edge strength histogram with 8 bins.Additionally, edge direction is computed at each pixel in the image toform a 16-bin histogram of these direction measures.

Lines

According to one embodiment, the line features comprise a line lengthhistogram. Preferably, an edge image is formed using the Sobeltransform. Preferably, lines are detected via application of the Houghtransform. Generally, the quantity of lines of different lengths isenumerated into a histogram with bins representing line lengths of 1 to3, 4 to 7, 8 to 15, 16 to 31, 32 to 64, and 64+ pixels.

Texture

According to one embodiment, the texture features comprise a “texton”histogram and statistics of a Gray-level Co-occurrence Matrix (GLCM).Preferably, the Leung-Malik filter bank is used, as described in T.Leung and J. Malik, Representing and Recognizing the Visual Appearanceof Materials Using Three Dimensional Textons, International Journal ofComputer Vision, 43:29-44 (2001), which is incorporated herein byreference as if set forth herein in its entirety, which consists ofedge, bar, and spot filters at different sizes and orientations.Generally, each filter is convolved with a given image, producing aresponse vector for each pixel in the image region. To form a set oftextons, these response vectors are clustered with k-means over a set of“training” images to produce clusters, with each cluster centerrepresenting a texton, as described in M. Varma and A. Zisserman, AStatistical Approach to Texture Classification from Single Images,International Journal of Computer Vision Special Issue on TextureAnalysis and Synthesis, 62(1-2):61-81 (2005), which is incorporatedherein by reference as if set forth herein in its entirety. As usedherein, “training” images, frames, or data are those that are used totrain classifiers (i.e. establish patterns and standards inclassifiers), such that classifiers are able to subsequently identifyand classify like image features (described in greater detail below).

Given a new image (i.e. a non-training image), the response vectors arecomputed and the Euclidean distance to each texton is computed to findthe closest match for each pixel in the image, thus assigning each pixelto a texton. Accordingly, a texton histogram is computed to provide thedistribution of textons within a given image region.

In one embodiment of the present system, the statistics of the GLCM arealso used as measures of texture. Generally, the GLCM is formed, and thestatistics comprising contrast, correlation, energy, entropy, andhomogeneity are computed, as described in C. C. Gotlieb and H. E.Kreyszig, Texture Descriptors Based on Co-Occurrence Matrices, ComputerVision, Graphics and Image Processing, 51:76-80 (1990); L. Lepisto etal., Comparison of Some Content-Based Image Retrieval Systems with RockTexture Images, In Proceedings of 10^(th) Finnish AI Conference, pp.156-63 (2002); and M. Partio et al., Rock Texture Retrieval Using GrayLevel Co-Occurrence Matrix, In 5^(th) Nordic Signal Processing Symposium(2002), all of which are incorporated herein by reference as if setforth herein in their entirety.

Shape

According to one embodiment, the shape features comprise circularity,convexity, polygon, and angularity features that characterize theboundary of an image region. Generally, circularity is defined as theratio of the area of a given image region to the area of a circle havingthe same perimeter, as represented by the following ratio:

$\frac{4{\Pi \cdot {area}}}{{perimeter}^{2}}$

and as described in V. Mikli et al., Characterization of Powder ParticleMorphology, In Proceedings of Estonian Academy of Sciences, Engineering,vol. 7, pp. 22-34 (2001), which is incorporated herein by reference asif set forth herein in its entirety. Convexity is generally computedusing the convex hull of an image region, as defined by the ratios:

$\frac{{perimeter}_{convexhull}}{{perimeter}_{region}},{and}$$\frac{{area}_{region}}{{area}_{convexhull}}$

and as described in M. Peura and J. Iivarinen, Efficiency of SimpleShape Descriptors, In Proceedings of the Third International Workshop onVisual Form, pp. 443-51 (1997), which is incorporated herein byreference as if set forth herein in its entirety. Typically, theboundary of an image region is fit to a polygon (i.e. a polygon isdetermined that best approximates the boundary of the image region to aspecified approximation accuracy), and the mean, standard deviation, andmaximum edge length of the polygon comprise another set of shapefeatures. Generally, angularity is computed as the standard deviation ofthe curvature at each boundary point, as described in J. Fox et al.,Onboard Autonomous Rock Shape Analysis for Mars Rovers, In IEEEAerospace Conference Proceedings (2002), which is incorporated herein byreference as if set forth in its entirety.

Given a region in a frame or image, either to be used as training data(described below) or desirous of classification, the results for each ofthe features (i.e. color, edges, lines, texture, shape, etc.) areconcatenated together to form a feature vector representation of theimage region. As used herein, a “feature vector” describes anN-dimensional vector of numerical features that represent the contentshown in an image or region of an image. As will be understood andappreciated by one of ordinary skill in the art, creation and use offeature vectors facilitates processing, analysis, and classification ofimages.

According to one embodiment, before the features are calculated on animage, the image is blurred with a Gaussian kernel of, preferably, size5×5 to reduce pixel noise within the image. As will be appreciated,while a size of 5×5 is preferred, other embodiments of the presentsystem use other sizes as will occur to one of ordinary skill in theart. Generally, both training images and images desirous ofclassification are blurred before calculating and forming featurevectors for the image. Additionally, in one embodiment, each feature inthe feature vector over a set of training data is normalized to fallbetween 0 and 1 by computing the maximum and minimum values of eachfeature and resealing the data. The same resealing is then used on anyfurther computed feature vectors.

Machine Learning Classifiers

As will be described below, several different classifiers are used inassociation with embodiments of the present system 10. As used herein, a“classifier” refers to an algorithm that, based on a set ofhuman-labeled training data, assigns classification scores to images orregions of images indentifying the probability that a given imagecontains a particular type of content. Classifiers are trained with setsof feature vectors extracted from training images that have beenhand-labeled as including a certain type of content. For example,hypothetical image region 320 shown in FIG. 3 would likely be labeled as“sky/clouds”, or something similar. Thus, assuming the frame 24 in FIG.3 is used as a training image, a feature vector containing featuresextracted from region 320 would be used as training data to train aclassifier to recognize similar content and identify it as “sky/clouds.”Once trained, a classifier is able to predict a label or labels for anew feature vector (i.e. a feature vector extracted and formed from animage desirous of classification) in the form of a classification scorefor each category of content on which the classifier has been trained.According to one embodiment, the resulting classification scores areinterpreted as probabilities that a given image includes thecorresponding content, and thus the classification scores for each typeof content for a given image sum to one. Additionally, in oneembodiment, multi-class classifiers are used, in which case the scoresalso sum to one.

Generally, two main types of known classifiers are preferred accordingto various embodiments of the present system: Support Vector Machine(SVM) classifiers and Random Forest classifiers. The preferred SVMtraining library is libSVM, as described in C.-C. Chang and C.-J. Lin,LIBSVM: A Library for Support Vector Machines, 2001, available athttp://www.csie.ntu.edu.tw/.about.cjlin/libsvm, which is incorporatedherein by reference as if set forth herein in its entirety, althoughother libraries and training data are possible. Generally, the higherthe quantity of training data used (i.e. the more training images used),the more accurate the results of a classifier become. Thus, preferably,a large library of training images are used for each classifierdiscussed herein. For example, the training library used in one test ofan embodiment of the present system (described below in the“Experimental Results” section) includes over 10,000 training images.Further, both linear and radial basis function kernels are used inassociation with various SVM classifiers as identified below. Accordingto one embodiment, Random Forests are used in a similar manner as thatdescribed in Shotten (2008) (cited previously), which is incorporatedherein by reference as if set forth herein in its entirety. As will beunderstood, while SVM and Random Forests classifiers are preferred,other types of classifiers are incorporated and used according tovarious embodiments of the present system.

Generally, the processes and functions described below presuppose thatone or more classifiers have been trained for each discrete process, andthe processes as described operate on a new image/frame (i.e. an imagedesirous of classification). Generally, a classifier is trainedaccording to the same procedures and processes as are used to identifyand classify new images. Accordingly, unless otherwise indicated, it isassumed that the procedures for training classifiers are similar toprocedures used for classification of new images, as described in detailbelow.

Intensity Classification

As described in reference to FIG. 1, after a frame has been extractedfrom a video file, it is analyzed via the intensity classificationprocess 400 to determine if it is a “dark” frame. If the content of agiven frame is too dark (e.g. the associated shot was filmed at night orwith inadequate lighting), then use of the frame for either training orclassification purposes should be avoided, as it tends to skew theresults. Accordingly, dark frames should be discarded (i.e. ignored)during the overall classification process 200.

FIG. 4 is a flowchart illustrating one embodiment of the intensityclassification process 400 for identifying “dark” or otherwise poorframes in a video file 22. Starting at step 405, a frame is received forclassification. The frame is then divided into N×N grid cells such thatclassification is achieved for smaller image regions, typically leadingto more accurate classification results (step 410). Preferably, eachframe is divided into a 4.times.4 grid, but various grid sizes are usedaccording to various embodiments of the present system 10. At steps 415and 420, a first grid cell of the frame is selected for processing, andthe intensity value is computed for each pixel in the selected cell(i.e. an intensity feature set) by converting the image (and cell) fromRGB colorspace to greyscale, wherein the greyscale image comprises theintensity image. Typically, the intensity feature set is computed as alinear combination of red, green, and blue values. The intensity valuesare then averaged across the cell to determine an average intensityvalue for the selected cell (step 425). At step 430, the systemdetermines whether any cells are remaining in the frame in which theintensity values have not been calculated. If cells are remaining, thenthe next cell is selected (step 415), and steps 420-430 are repeated forthe new cell.

After all cells in the given frame have been processed (i.e. the averageintensity value has been calculated for each cell), the averageintensity values are concatenated to form an intensity feature vector ofN² values for the frame (step 435). For example, if the preferred4.times.4 grid size is used, then the resulting intensity feature vectorwill include 16 elements/values. Once the intensity feature vector isformed, the vector is classified via an intensity classifier todetermine if the corresponding image is “dark” or not (step 440).Generally, a predefined threshold value is selected by a system operatordepending on the level of darkness the operator is willing to accept,and the classification score produced during classification (step 440)is compared to the threshold. If the classification score exceeds thethreshold, then the frame is deemed a “dark” frame, and is discarded. Ifthe frame is not a dark frame, then the frame is processed further (seeFIG. 2).

For purposes of training a classifier used for intensity classification,steps 405-435 are repeated for each training image. Each training imageis hand-labeled by a system operator as “dark” or “not dark,” and theresulting intensity feature vectors are associated with these labels.The labels and associated feature vectors are used to train a SVMclassifier with a linear kernel (typically, a linear kernel ispreferable when the classification problem is approximately linearlyseparable, as is the case here). Thus, given a new image, intensityclassification process 400 is able to classify the image accordingly as“dark” or “not dark.”

Indoor/Outdoor Classification

Once it has been determined that a given frame is not a dark frame, andis in fact an acceptable frame for classification purposes, the frame isfurther analyzed via the indoor/outdoor classification process 500 todetermine if the frame is an “outdoor” frame, an “indoor” frame, or an“undetermined” frame. According to one embodiment of the present system,shots and/or frames including content of indoor or undetermined scenesare classified as such, but no further analysis or sub-classification isperformed on the frames. Thus, if a given frame is classified as anoutdoor frame, the frame is further classified (as described below)based on distinct categories of outdoor scenes (i.e. scene classes). Aswill be understood and appreciated, however, embodiments of the presentsystem are not limited to outdoor scenes, and are capable of identifyingand classifying varying types of indoor scenes depending on types oftraining data and features used. For purposes of illustration, however,an exemplary embodiment for classifying categories of outdoor scenes isdescribed, but is not intended to limit the present system in any way.

FIG. 5 is a flowchart illustrating an embodiment of the indoor/outdoorclassification process 500 for classifying frames as “indoor,”“outdoor,” or “undetermined.” As used herein, “indoor” frames are thosethat include content depicting the inside of buildings or structures(e.g. offices, houses, hospitals, etc.). As used herein, “outdoor”frames are those that include content depicting outdoor scenes (e.g.deserts, mountains, the outside of buildings, roads, etc.). As also usedherein, “undetermined” frames are those that include content that doesnot clearly depict either an indoor or outdoor scene (e.g. close-upviews of characters, actors, or objects in which little or no backgroundis visible, blurry frames, etc.).

Starting at step 505, a frame is received for classification. The frameis then divided into N.times.N grid cells such that classification isachieved for smaller image regions, typically leading to more accurateclassification results (step 510). Preferably, each frame is dividedinto a 4.times.4 grid, but various grid sizes are used according tovarious embodiments of the present system 10. At steps 515 and 520, afirst grid cell of the frame is selected for processing, and the color,edge, line, and texture features (described previously) are calculatedfor the given cell to form a feature vector for the cell. Because eachgrid cell is a rectangular portion of the frame, the shape features arenot calculated (i.e. the rectangular shape is already known). As will beunderstood, other features in addition to those described are used invarious embodiments of the present system as will occur to one ofordinary skill in the art.

At step 525, the feature vector for the selected cell is classified viaa classifier to determine the corresponding class for the cell (i.e.indoor, outdoor, or undetermined). For purposes of training thisclassifier, steps 505-520 are repeated for each cell in each trainingimage. Each cell in each training image is hand-labeled by a systemoperator as “indoor,” “outdoor,” or “undetermined,” and the resultingfeature vectors for each cell are associated with these labels. Thelabels and associated feature vectors are used to train a SVM classifierwith a radial basis function kernel. Typically, a radial basis functionkernel is preferable for this classifier because a more complex modelgenerally produces more accurate classification results. The resultingclassification vector for the cell generally comprises a3.times.1-dimensional vector, wherein the 3 values/elements in eachvector comprise the classification scores (between 0 and 1) for each ofthe three possible classes (i.e. indoor, outdoor, or undetermined). Atstep 530, the system determines whether any unclassified cells areremaining in the frame. If cells are remaining, then the next cell isselected (step 515), and steps 520-530 are repeated for the new cell.

After the classification vectors for all cells in the given frame havebeen calculated (via step 525), the classification vectors areconcatenated to form an indoor/outdoor feature vector for the overallframe that includes the classification scores for each cell (step 535).Once this indoor/outdoor feature vector is formed, the vector isclassified via an indoor/outdoor classifier to determine if thecorresponding frame is an indoor, outdoor, or undetermined frame (step540). The classifier used in step 540 is trained based on indoor/outdoorfeature vectors associated with training images that are labeled by asystem operator as indoor, outdoor, or undetermined frames. Just as withthe classifier associated with step 525, the classifier used in step 540is a SVM classifier; however, in this case, a linear kernel is selectedbecause the data is approximately linearly separable, and the selectionof a linear kernel prevents over-fitting as could occur with the use ofa radial basis function kernel. Generally, for a new image (i.e.non-training image), a classification score is calculated during step540 for each of the three classes associated with the classifier (i.e.indoor, outdoor, and undetermined). Typically, the highestclassification score of the three is the type of content most likelyassociated with the frame, and the frame is labeled accordingly.According to the presently-described embodiment, if the frame is labeledan “outdoor” frame, then it is processed further via the outdoorclassification process 600. Otherwise, the frame is assigned an overallclassification score of “0” for all outdoor classes and stored in adatabase 14 for subsequent processing (see step 235 in FIG. 2 andassociated discussion).

Still referring to the embodiment of the indoor/outdoor classificationprocess 500 described in FIG. 5, as described, two separate classifiersare used (i.e. classifiers associated with steps 525 and 540) forpurposes of, typically, producing more accurate classification results,as such classifiers take into account variations in content acrossdifferent portions of a frame. It is understood, however, thatembodiments of the present system 10 are not limited to a two-classifierapproach, and a one-classifier approach may be used (i.e. training aclassifier on raw features extracted from an entire image, as opposed todividing the image into cells) if a system operator is content with(typically) less accurate results.

Outdoor Classification

After a frame has been labeled as an outdoor frame, the frame isanalyzed by the outdoor classification process 600 to determine whichcategory or categories of material class(es) (if any) apply to theframe. As described previously and as used herein, “material” or“material class” refers to the type of physical content shown in aframe. According to one embodiment of the present system, materialsinclude building (i.e. the outside of a structure), grass, person,road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice,trees/plants, vehicle, water, and miscellaneous. As will be understood,however, embodiments of the present system are not limited to theparticular material classes described, and other similar classes areused according to various embodiments of the present system. Asmentioned previously, once each frame in a given video file 22 orportion of a video file has been classified, the material class resultsare aggregated and averaged via the video file classification process1000, 1001 to identify one or more scene classes for each video file orportion thereof.

Referring now to FIG. 6, a flowchart is shown illustrating the steps andfunctions involved in the outdoor frame classification process 600according to one embodiment of the present system 10. Generally, theprocess 600 involves receiving a frame from a video file 22 andsegmenting the frame into regions of similar content based on apredetermined segmentation algorithm. Preferably, because segmentationalgorithms tend to produce varying results, each frame is segmentedmultiple times based on multiple sets of parameters (described below).The features from each region for each segmentation are extracted, andeach region is classified by material using a hierarchy of SVMclassifiers. The material classification results are combined across themultiple segmentations by averaging the classification scores for eachmaterial for each pixel in the region. The combination of classificationscores results in, for each pixel in the frame, a score for eachmaterial category (that sum to 1 for each pixel), with the largest scorerepresenting the material category that is most likely associated witheach pixel. From these material scores, a material arrangement vector isgenerated describing the spatial relation of material(s) in a givenframe. The material arrangement vector is used to classify the framebased on zero or more scene categories/classes (described below).

Segmentation

As shown in FIG. 6, starting at step 605, a frame is received forclassification. At step 610, the frame is divided into segments (i.e.segmented) based on the content shown in each segment. Each separatesegment generally comprises an image region with relatively uniformcolor and texture. The underlying goal of segmentation is to identifyand separate image regions that contain different types of content (e.g.different materials) to produce more accurate classification results forthe image, and subsequently the video file. According to one embodiment,the segmentation algorithm used is Efficient Graph-Based Segmentation,as described in P. F. Felzenszwalb and D. P. Huttenlocher, EfficientGraph-Based Image Segmentation, International Journal of ComputerVision, vol. 59, no. 2 (2004), which is incorporated herein by referenceas if set forth herein in its entirety.

Because segmentation algorithms tend to produce varying results based onthe parameters used, multiple segmentations are calculated for eachframe according to one embodiment, as suggested in D. Hoiem et al.,Geometric Context from a Single Image, International Conference ofComputer Vision (ICCV), IEEE, vol. 1, pp. 654-61 (2005); and G. Mori etal., Recovering Human Body Configurations Combining Segmentation andRecognition, In IEEE Computer Vision and Pattern Recognition (2004),both of which are incorporated herein by reference as if set forthherein in their entirety. Preferably, three different segmentations arecomputed for each frame. Thus, according to a preferred embodiment, foreach frame extracted from a video file, three different parameter setsare used in the Efficient Graph-Based Segmentation algorithm, namely.sigma.=0.325, k=500; .sigma.=0.4, k=180; and .sigma.=0.5, k=160, with aminimum segment size of 500 pixels, wherein u is used to smooth theimage before segmenting it, and k comprises a value for the thresholdfunction. As will be understood, however, embodiments of the presentsystem are not limited by these particular parameters, nor by use ofonly three segmentations, and other parameters and multiples ofsegmentations are used according to various embodiments.

As will be appreciated, some of the segments produced via thesegmentation algorithm (step 610) are small, or may comprise only partof a larger object or region of material. Accordingly, in oneembodiment, in order to achieve higher accuracy and faster computationspeeds during subsequent material classification, segments includingsimilar classes of materials are merged together. FIG. 7 is a flowchartillustrating an embodiment of the segment combination/merging process700 for combining like segments in a frame. As mentioned previously,each frame is typically segmented multiple times, producing multiplesets of segments (i.e. segmentations). Thus, as will be understood byone of ordinary skill in the art, the combination/merging process 700 isrepeated for each discrete set of segments for each frame (preferably,three sets corresponding to three segmentations).

Starting at step 705, the features are extracted/calculated from eachsegment to form a feature vector for each segment. According to apreferred embodiment, the extracted features correspond to thosementioned previously (i.e. color, edge, line, texture, and shape), butother features are used in other embodiments. The extracted features areconcatenated into a feature vector for the segment. At step 710, anaffinity score is calculated for each pair of adjacent segments. As usedherein, an “affinity score” is the result/score from an adjacencyclassifier predicting whether two adjacent segments comprise or belongto the same material class. According to one embodiment, the adjacencyclassifier comprises a Random Forest classifier that operates on theabsolute value of the difference between feature vectors of adjacentsegments. Preferably, a Random Forest classifier is used (as opposed toa SVM) classifier to improve computation speed. Generally, in order totrain the classifier, the feature vectors of a plurality of adjacentsegments in a plurality of training images are compared, and theabsolute value of the difference is calculated for each and used as thetraining feature set for the classifier. Each absolute value vector islabeled by a system operator as a positive result (i.e. the adjacentsegments correspond to the same material class) or a negative result(i.e. the adjacent segments correspond to different material classes).Thus, given a pair of adjacent segments from a new frame (i.e. a framedesirous of classification), the affinity score produced by theadjacency classifier of step 710 represents the probability that the twosegments include content associated with the same class.

Still referring to FIG. 7, for a new frame, once all affinity scoreshave been calculated for all pairs of segments in a given segmentation,the highest affinity score is identified (step 715). If the highestaffinity score is above a predefined threshold (step 720), then the pairof segments associated with the affinity score are merged into a singlesegment (step 730). Once merged, the feature vector of the mergedsegment is recalculated (step 735). Then, based on the newly-definedsegment and adjacent segments, the affinity scores are recalculated foreach pair of adjacent segments (step 710).

Steps 710-735 are repeated until the highest affinity score is no longerabove the predefined threshold (i.e. greedy, hill-climbing combination).When this occurs, the most recent set of segment feature vectors(calculated at step 735) are stored in a database 14 for subsequentprocessing. As will be understood, the segments in some frames do notrequire combination because they are adequately and accurately segmentedduring the segmentation step 610 (see FIG. 6). Accordingly, the segmentfeature vectors initially calculated for these segments at step 705 arestored in the database via step 725, and no further merging orrecalculating is necessary.

According to one embodiment, to determine an appropriate affinity scorethreshold value, the adjacency classifier is calibrated with avalidation set of images/frames to produce a desired accuracy. As willbe understood, a “validation set” refers to a set of images used to testa classifier that have been labeled by a system operator such that theactual class of each image is known. To determine an appropriatethreshold value, a system operator selects an arbitrary value andperforms process 700 on a set of validation frames. Because the actualclass of each segment is known, the precision value of correct segmentcombinations can be calculated (i.e. the proportion of combined segmentsthat actually belong to the same class as compared to all combinedsegments). If the precision is less than the desired precision (e.g.97%), then the affinity score threshold should be increased (and viceversa). This process should be repeated until a desired precision isreached.

Again, as will be understood, segment combination/merging process 700 iscompleted for each separate segmentation for each frame. Thus, forexample, if a given frame is segmented three times, process 700 isrepeated for each of the three segmentations, and the results of eachare stored in a database 14.

Material Classification

Referring again to FIG. 6, after each segment has been defined for eachof the multiple segmentations for a given frame via steps 605, 610, andthe segment combination/merging process 700, the segment feature vectorsare used to classify each segment (and subsequently each associatedframe) based on a set of predefined material classes (step 615). Beforeeach segment can be classified, however, a set of material classifiersassociated with the predefined classes are developed using a pluralityof training images. For training purposes, the segment feature vectorsfor each segment associated with each training image are formed asdescribed previously in association with segment step 610 andcombination process 700. These segment feature vectors are labeledaccording to their respective material classes by a system operator, andthe labeled vectors are used to train a hierarchy of SVM classifiers(described in further detail below) (see FIG. 8 for exemplary classifierhierarchy).

According to one embodiment, libSVM (mentioned previously) is used toform a one-to-one classifier for each pair of material classes andproduce a classification result as a combination of these classifiers.Generally, each one-to-one classifier comprises a SVM classifier with aradial basis function kernel. Use of such a combination of one-to-oneclassifiers is conventional for a multi-class problem (i.e. whenmultiple classes potentially apply to a single image region). For Nclasses, the number of classifiers required is defined by:

$\frac{N\left( {N - 1} \right)}{2}$

For example, for an embodiment that includes 12 material classes (e.g.building, grass, person, road/sidewalk, rock, sand/gravel/soil,sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous),66 classifiers are used to accurately classify each region (i.e.12(12-1)/2=66). This large number of classifiers is required becauseeach class must be compared against every other class to achieve acomplete result.

According to a preferred embodiment, however, rather than a conventionalone-to-one classifier arrangement, a hierarchy of classifiers is used.The hierarchy utilizes one-to-one classifiers, but based onpredetermined knowledge about the classes (i.e. they are explicitlypredefined to correspond to materials), a more effective arrangement ofone-to-one classifiers is constructed. FIG. 8 shows an exemplaryhierarchical tree representing an organization of classifiers 800 usedto classify materials. As shown, each node/box 805 represents a materialclass or category. The classes are pre-organized based on inherentknowledge about the content of the classes, thus requiring fewerclassifiers and computational steps to classify a segment or imageregion. For example, the categories of “building,” “person,” and“vehicle” are organized under the “man-made” category. This type oforganization presupposes that, based on the particular type of imagecontent, not every class need be compared to every other class. Forexample, this hierarchy assumes that “vehicle” content does not requirecomparison to “grass” content, etc. As will be understood, eachhierarchy of classifiers is predetermined by a system operator based onthe classes of materials used, and the materials and arrangement shownin FIG. 8 are presented for exemplary purposes only, and are notintended to limit the present system 10 in any way.

According to the hierarchical type of arrangement shown in FIG. 8, whenonly two child classes extend from a parent class, a single one-to-oneclassifier is sufficient to achieve an accurate result between the twochild classes (as defined by the classifier equation above, i.e.2(2-1)/2=1). For example, the “grass” and “trees” classes in FIG. 8 onlyuse a single one-to-one classifier. When three children exist, threeone-to-one classifiers are used (i.e. 3(3-1)/2=3), and so on. For parentclasses, training data from all child classes are used to train theassociated classifiers (e.g. “building,” “person,” and “vehicle” data isused to train the “man-made” classifier). Generally, better accuracy andfaster computational speeds are achieved using a hierarchical approachas it makes effective use of known data types.

For a hierarchy or tree of classifiers 800 (such as that shown in FIG.8), a classification result for each segment or image region is producedby multiplying the results of each intermediate classifier down thetree. Specifically, assume CX (f) represents a given classifier result,in the form of a vector comprising a score for each material class, on anode x in the tree and a feature vector f. For leaf nodes (i.e. thenodes with no child nodes extending therefrom, such as “person,” “sky,”“road,” “rock,” etc.), the classifier result is defined by:

${C_{x}\left( \underset{f}{\rightarrow} \right)}_{i} = \left\{ \begin{matrix}{0,{{material} \neq x}} \\{1,{{material} = x}}\end{matrix} \right.$

For non-leaf nodes (i.e. those nodes with child nodes extendingtherefrom, such as “man-made,” “vegetation,” etc.), the classifierresult is defined by:

${{C_{x}\left( \underset{f}{\rightarrow} \right)} = {\sum\limits_{y \in {{children}{(x)}}}{{c_{x}\left( \underset{f}{\rightarrow} \right)}_{y}{C_{y}\left( \underset{f}{\rightarrow} \right)}}}},$

where C_(x) ({right arrow over (f)}) represents the result of the SVMclassifier for given node x, and c_(x) ({right arrow over (f)})_(y) isthe result for given class y. For example, given a segment to beclassified, the classification score for the segment for the “building”class comprises the result of the material classifier for “man-made”multiplied by the result of the man-made classifier for “building.”

For a new frame (i.e. a frame desirous of classification), thepreviously-calculated segment feature vectors associated with the frameare retrieved from a database 14 (see FIG. 7 and associated discussionfor calculation and storage of segment feature vectors), and processedvia a hierarchical material classification tree, such as that shown inFIG. 8 (step 615). Generally, the scores produced for each SVMclassifier within the tree sum to 1, and thus the scores produced by thetree for each included class sums to 1, with each material scorecomprising a value between 0 and 1. The material with the greatest score(i.e. highest decimal value between 0 and 1) represents the materialtype most likely contained within a given segment. Thus, after beingclassified by a hierarchical set of material classifiers, each segmentis associated with a vector of material scores, wherein the vectorincludes a value between 0 and 1 for each predefined material class.

Again referring to FIG. 6, based on the vectors of material scores, thematerial scores for each pixel in a given segment are calculated (step620). According to one embodiment, the material scores associated with agiven segment are assigned to each pixel in the segment. If multiplesegmentations are used, the material values for each segmentation areaveraged at each pixel to produce a vector of material scores for eachpixel. Thus, each pixel is represented by a vector of material scores ofN.times.1 dimension, wherein N represents the number of classes, andeach material value in the vector represents an average of that materialscore across each segmentation of the frame. At step 625, the materialscores associated with each pixel are stored in a database 14 forsubsequent processing.

Material Arrangement Vector

Referring now to FIG. 9, a flowchart is shown illustrating the stepsassociated with an embodiment of the material arrangement vectorgeneration process 900. Generally, a material arrangement vector definesor characterizes the spatial arrangement of material content in a givenframe based on previously-calculated material scores for pixels withinthe frame. To compute a material arrangement vector for a frame, theframe is first divided into N×N grid cells (step 905). As will beunderstood, as grid sizes become progressively more finite (e.g. 1×1,2×2, 4×4, etc.), the material vector comprises a more accuraterepresentation of the spatial arrangement of materials in a frame. Forexample, if a 1×1 grid size is used, then a resulting materialarrangement vector is identical to a material occurrence vector (i.e.the vector describing proportion or occurrence of materials in a givencell or region) for the entire frame, thus indicating the proportion(but not spatial arrangement) of each material in the frame. Asprogressively more detailed grid sizes are used (e.g. 2×2, 4×4, 8×8,etc.), the spatial arrangement of the materials in the image becomesclearer as each grid cell defines the specific material type(s)contained in the cell, and the resulting class of material for eachregion in the image becomes known.

Regardless of the grid size used, at step 910, a cell is selected forprocessing. At step 915, the material scores (i.e. vector of materialscores) for each pixel in the selected cell are retrieved from adatabase 14. The vectors of material scores for each pixel in the cellare averaged to produce a material occurrence vector for the cell (step920). As described, the material occurrence vector identifies thetype(s) of material likely contained in the cell based on the materialscore for each class of material in the vector. At step 925, the systemdetermines whether any unprocessed cells are remaining in the frame. Ifso, steps 910-925 are repeated for the next cell. Once the materialoccurrence vectors have been calculated for all cells in the frame, theoccurrence vectors are concatenated to form the material arrangementvector for the frame (step 930). According to one embodiment, materialarrangement vector generation process 900 is repeated for a given frameusing many different grid sizes, and the resulting material arrangementvectors are used to train varying classifiers, whereby theclassification results are averaged to produce more accurate sceneclassifications.

Scene Classifiers

Referring again to FIG. 6, at step 630, a given frame is classifiedbased on the material arrangement vector for the frame (i.e. a sceneclassification score vector is calculated for the frame). Each frame isassociated with a scene classification score vector comprisingclassification scores for predefined classes of scenes (e.g.coast/beach, desert, forest, grassland, highway, indoor, lake/river,mountainous, open water, outdoor, sky, snow, urban, etc.). Before such ascene classification score vector can be calculated, however, aclassifier must be trained to generate such vectors. Two embodiments ofscene classifiers are described below.

Proportional Classifiers

According to one embodiment of the present system 10, a proportionalclassifier operates on material occurrence vectors (i.e. materialarrangement vectors associated with 1.times.1 grid sizes). For aplurality of training images/frames, the material arrangement vectorsare calculated for a 1.times.1 grid size according to process 900. Thesevectors are labeled by a system operator according to the sceneclass(es) associated with the corresponding frames (based on content).According to one embodiment, more than one scene class may apply to agiven frame. Alternatively, some frames include no defined sceneclasses, and are labeled as such. In one embodiment, each training imageis flipped horizontally and the material arrangement vector isrecalculated to provide additional training data to each classifier. Foreach scene type (e.g. coast/beach, desert, etc.), a SVM classifier witha radial basis function kernel is trained based on material arrangementvectors associated with that scene type. Given a new frame, each sceneclassifier classifies the material arrangement vector associated withthe frame to determine a classification score (between 0 and 1) for theframe (i.e. the higher the score, the more likely it is the frameincludes that class of content). These classification scores arecollected into a scene classification score vector and stored in adatabase 14 for further video file classification (step 235, see FIG.2).

Spatial Pyramid Classifiers

According to a preferred embodiment, a spatial pyramid classifier isused to classify frames according to scene types. Examples of spatialpyramids are described in Lazebnik (2006) (cited previously), which isincorporated herein by reference as if set forth herein in its entirety.The spatial pyramid classifiers operate in much the same way as theproportional classifiers (described above), except that each type (i.e.scene class) of classifier is trained using material arrangement vectorsassociated with varying grid sizes, and the results are combined foreach type. Specifically, material arrangement vectors are calculated foreach training frame according to process 900 for multiple grid sizes(e.g. 1×1, 2×2, 4×4, etc.). For each grid size, a separate classifier istrained using the resultant material arrangement vectors from thetraining images for that grid size for each scene type. Accordingly,each scene type includes not one, but a multiple number of classifierscorresponding to multiple grid sizes. For example, if three grid sizesare used for each frame (e.g. 1×1, 2×2, 4×4), then each scene typeincludes three classifiers. Again, each material arrangement vector foreach training frame is labeled by hand by a system operator. Also,according to one embodiment, each training image is flipped horizontallyand the material arrangement vector is recalculated to provideadditional training data to each classifier.

For a new frame (i.e. a frame desirous of classification), a materialarrangement vector is calculated for each grid size in which theclassifiers have been trained. Thus, during scene classification (step630), a scene classification score vector is generated for each gridsize for each frame. According to one embodiment, a weighted sum of thescene classification score vectors is produced to define a sceneclassification score vector for the frame. For example, if threedifferent grid sizes are used corresponding to 1×1, 2×2, and 4×4 gridcells, the results for each size are weighted and combined (e.g.weighting of 0.25 for 1×1, 0.25 for 2×2, and 0.5 for 4×4). Thus, thescene classification score values for the 1×1 grid size are multipliedby 0.25, the values for the 2×2 grid size are multiplied by 0.25, andthe values for the 4×4 grid size are multiplied by 0.5, whereby theresulting weighted values are added together to produce a sceneclassification score vector for the given frame. Generally, a higherweight is associated with higher grid sizes because those sizes aretypically more accurate (although this is not always the case). As willbe understood and appreciated by one of ordinary skill in the art, avariety of multiples of grid sizes, number of grid cells used, andweights associated with the grid sizes are used according to variousembodiments of the present system.

Video File Classification

Referring again to FIG. 2, after the scene classification scores (i.e.scene classification score vectors) have been calculated for eachprocessed frame in the video file, the video file itself is classifiedaccording to one embodiment of the video file classification processes1000, 1001. Generally, two different types of video file classificationprocesses 1000, 1001 are used depending on the type of video filereceived, as well as the purpose of the classification.

Predefined Shot

FIG. 10A is a flowchart illustrating the steps involved in oneembodiment of the video file classification process 1000 for apredefined shot. Classification process 1000 is used when the receivedvideo file comprises a predefined shot, or a sequence of predefinedshots with clearly defined start and end timecodes, or an entire videowith clearly defined start and end timecodes. Thus, the overall purposeof process 1000 is to classify a portion of video that is alreadydivided into a discrete unit (e.g. via the process described in Rasheed(2003), cited previously). As will be understood, if a video fileincludes more than one discrete unit (e.g. a plurality of shots), eachof the units is analyzed and classified separately, providing aclassification score/result for each separate unit.

Starting at step 1005, the scene classification score vectors for eachframe in the video file, shot, or other discrete unit of video areretrieved from the database 14. For frames that were classified as“indoor” or “undetermined,” the classification score of “0” for outdoorclasses (i.e. a vector of zero values corresponding to each outdoorscene class) is retrieved for those frames. At step 1010, theclassification scores for each scene class for each frame are averagedacross all frames in the given unit of video (i.e. shot). For “indoor”or “undetermined” frames, the “0” value is used in the averagecalculation for each outdoor scene class for that particular frame, thuslowering the overall average for the shot. The average scene classscores produce a final classification score for each scene class for theshot (an example of which is shown in table 26 a in FIG. 1). As shown inexemplary output 26 a, a classification score is provided for each sceneclass, with the higher scores indicating a higher likelihood that agiven shot includes content associated with the identified scene class.As will be understood and appreciated, rather than averaging the sceneclassification scores, the scores are analyzed in other intelligent waysaccording to various embodiments of the present system, such asexamining a median or maximum value, using signal processing across anentire video file, etc.

Generally, a predefined threshold value is set by a system operator foreach scene class, and any class with a classification score exceedingthat threshold is deemed as applying to the shot (step 1015). Accordingto one embodiment, the threshold value is determined on a per-classbasis (because different classes often perform differently based on thetype of classified content), and such thresholds are determined as afunction of precision and recall accuracy experiments using validationdata. Once the class(es) with classification scores exceeding thethreshold are identified, the shot is labeled according to theidentified scene classes (step 1020). If none of the classificationscores exceed a threshold, then no defined scene classes are associatedwith the shot (likely indicating the shot comprises some otherundetermined content). The classification results are then stored in adatabase 14 for subsequent purposes (step 1025), including generatingreports (step 245, see FIG. 2), indexing and retrieval, and othersubsequent purposes as will occur to one of ordinary skill in the art.

Shot Detection

According to one embodiment, rather than labeling predefined shots orsequences of video, the video classification process 1001 is used todetect shot breaks in a video file or sequence of frames. Accordingly,FIG. 10B is a flowchart illustrating the steps involved in oneembodiment of the video file classification process 1000 for shotdetection. Starting at step 1006 the scene classification score vectorsfor each frame in the video file are retrieved from the database 14. Atstep 1011, a classification score vector for a given frame is selectedfrom the retrieved set. Typically, the first vector selected is thevector associated with the first processed frame in the file, althoughthis is not necessarily the case. Regardless of which vector isselected, at step 1016, a classification score vector associated with asubsequent frame in the video file is selected. Typically, thesubsequent classification score vector will be for the next processedframe in the sequence of frames in the video file, although this is notnecessarily the case.

At step 1021, the absolute value of the difference between the sceneclass scores in the two selected classification score vectors arecalculated. For example, if the vector for the first selected frameincludes a classification score for class “forest” of 0.11, and thevector for the second selected frame includes a classification score forclass “forest” of 0.13, then the absolute value of the difference wouldbe 0.02. If this difference is above a predetermined threshold, then ashot break is identified between the two frames (step 1026, 1031). Theabsolute value of the difference is calculated for each scene class foreach vector, and each difference is compared to a predefined threshold.Typically, a large difference in classification scores between twoframes indicates a change in content between the two frames, andaccordingly, a shot break. If the difference is below a predefinedthreshold value, then no shot break is identified, and the systemdetermines whether any frames remain in the video file (step 1036). Ifframes are remaining, then a classification score vector associated witha subsequent frame in the video file sequence is selected and comparedto a vector for a previously-selected frame that precedes it.Accordingly, steps 1016, 1021, 1026, 1031, and 1036 are repeated untilall scene classification score vectors associated with a given videofile have been analyzed. Once all frames have been analyzed, allidentified shot breaks (if any) are stored in a database 14 for furtherreporting (e.g. table 26 b) and processing purposes.

As will be understood and as mentioned previously, the particular sceneclasses identified in output 26 a and listed herein are presented forillustrative purposes only, and are in no way intended to limit thescope of the present systems and methods. Additionally, the exemplaryoutputs 26 a, 26 b are presented for purposes of illustration only, andother types and formats of outputs and reports are generated accordingto various embodiments of the present system.

Referring now to FIG. 11, a representation of the system components 1100is shown according to one embodiment of the video classification system10. According to one embodiment, the previously-described processes andfunctions of the video classification system 10 are performed by theinternal system components/modules 1100 shown in FIG. 11. As shown, thesystem modules include an intensity classification module 1105, anindoor/outdoor classification module 1110, an outdoor classificationmodule 1115, a segmentation module 1120, a material arrangement module1125, and a video file classification module 1130. As will be understoodand appreciated, the components or modules shown in FIG. 11 arepresented for illustrative purposes only, and are not intended to limitthe scope of the present systems or methods in any way.

Experimental Results

To demonstrate functional capability, an embodiment of the presentsystem was tested to determine its classification performance andaccuracy. The embodiment tested was configured to detect and classifyvideo content according to outdoor material and scene classes asdescribed above. The video content and associated images used to testthe embodiment were obtained from the LabelMe database, as described inB. C. Russell et al., LabelMe: A Database and Web-Based Tool for ImageAnnotation, International Journal of Computer Vision, vol. 77, pp.157-73 (2008), which is incorporated herein by reference as if set forthin its entirety, as well as from Google®, Images, Flickr®, movies suchas Along Came Polly, Babel, Cheaper by the Dozen, Tears of the Sun, andWild Hogs, and an episode of the television program Lost.

Material Classification Results

For the test, 1019 images (i.e. frames) were extracted from theabove-referenced image and video databases, movies, and televisionprogram. Five-fold cross-validation was used to test the images, inwhich 80% of the images are used as training data and 20% are used asvalidation data (i.e. used to test the results). This process wasperformed five times until all images had been used as validation data,and the results were averaged over the five tests. The images wereprocessed and segmented as described above (see FIGS. 2-7 and associateddiscussion), and each segment was hand-labeled as one of the materialsshown in the confusion matrix 1200 in FIG. 12 (i.e. building, grass,person, road/sidewalk, rock, sand/gravel, sky/clouds, snow/ice,trees/bushes, vehicle, water, and miscellaneous).

The confusion matrix 1200 demonstrates the percentage of time that aregion labeled by the tested embodiment of the system as a givenmaterial was correctly labeled as that material as compared to thehand-labeled region, as well as the percentage of time the testedembodiment incorrectly classified a region as another material. Forexample, as shown in the confusion matrix, the tested embodimentcorrectly labeled image regions/segments that included content ofbuildings as “buildings” 69% of the time. As shown, the most accuratelyclassified material was “sky/clouds” (i.e. correctly classified 95% ofthe time), and the most common misclassification was “snow/ice,” whichwas incorrectly classified as “water” 25% of the time. By analyzing aconfusion matrix and adjusting threshold values, a system operator isable to customize the results based on his or her performancerequirements.

Scene Classification Results (Individual Images)

For the test, 10017 images (i.e. frames) were extracted from theabove-referenced image and video databases, movies, and televisionprogram. Five-fold cross-validation was again used to test the images,in which 80% of the images are used as training data and 20% are used asvalidation data (i.e. used to test the results). This process wasperformed five times until all images had been used as validation data,and the results were averaged over the five tests. The images wereprocessed as described above (see FIGS. 2-9 and associated discussion).Each image was hand-labeled according to a corresponding scene class orclasses, as listed in FIG. 12 (i.e. urban, coast/beach, desert, forest,grassland, highway, lake/river, mountainous, sky, snow, open water,indoor, and outdoor). Additionally, each image was labeled according tomore than one class if more than one type of scene content was presentin the image.

As used herein, “precision” represents the percentage of correctlyclassified images from all classified images (i.e. the fraction ofdetections that are true positives rather than false positives). As usedherein, “recall” represents the percentage of correctly classifiedimages from all images (i.e. the fraction of true labels that aredetected rather than missed). The precision-recall curve 1300 shown inFIG. 13 represents the plotted precision and recall results for eachscene class as scene classification threshold values are varied between0 and 1 in increments of 0.01 (i.e. this demonstrates the trade-offbetween accuracy and noise). Each point in the curve 1300 represents thecomputed precision and recall scores for a particular threshold value.

As shown in FIG. 13, of the outdoor classes, the “urban” categoryachieved the greatest accuracy. This is likely due, in part, to thelarge volume of training images available for the “urban” category. Thisresult demonstrates that when many training images are used, embodimentsof the present system are capable of classifying images with highaccuracy. As also shown, the “lake/river” category produced the poorestresults. This result is somewhat expected, however, because reflectionsof surrounding terrain onto small bodies of water in images oftenproduces confusing colors and spatial arrangements, creating difficultyin classification.

The foregoing description of the exemplary embodiments has beenpresented only for the purposes of illustration and description and isnot intended to be exhaustive or to limit the inventions to the preciseforms disclosed. Many modifications and variations are possible in lightof the above teaching.

The embodiments were chosen and described in order to explain theprinciples of the inventions and their practical application so as toenable others skilled in the art to utilize the inventions and variousembodiments and with various modifications as are suited to theparticular use contemplated. Alternative embodiments will becomeapparent to those skilled in the art to which the present inventionspertain without departing from their spirit and scope. Accordingly, thescope of the present inventions is defined by the appended claims ratherthan the foregoing description and the exemplary embodiments describedtherein.

We claim:
 1. A method for classifying videos based on video content,comprising the steps of: receiving a video file, the video fileincluding a plurality of frames, where each frame includes a pluralityof pixels; extracting a set of frames from the video file; for eachframe in the extracted set of frames, determining whether the framecomprises a poor classification frame, removing the one or more poorclassification frames from the extracted set of frames; dividing eachframe in the extracted set of frames into one or more segments, whereeach segment includes relatively uniform image content; extracting imagefeatures from each segment to form a feature vector associated with eachsegment; generating a material classification score vector for eachsegment via one or more material classifiers based on the feature vectorassociated with each segment, where each material classification scorevector includes one or more material classification scores associatedwith one or more predefined material content categories; and assigningeach material classification score vector associated with its respectivesegment to each pixel in each respective segment for each respectiveframe in the set of frames.
 2. The method of claim 1, further comprisingthe step of storing the material classification score vectors assignedto each pixel in a database for subsequent use in video fileclassifications.
 3. The method of claim 1, further comprising the stepof combining adjacent segments based on similar image content featuresextracted from the segments.
 4. The method of claim 1, where poorclassification frames are determined via one or more classifiers.
 5. Themethod of claim 1, where a poor classification frame comprises a frameassociated with at least one of the following frame types: a frame shotin low light, a frame shot at night, a blurry frame, and an undeterminedframe.
 6. The method of claim 1, where image features comprise one ormore features selected from the group comprising: color features, edgefeatures, texture features, and shape features.
 7. The method of claim1, where image features comprise data associated with image content. 8.The method of claim 1, where one or more material classifiers arehierarchically related.
 9. The method of claim 1, where the one or morepredefined material content categories are selected from the groupcomprising: building, grass, person, road/sidewalk, rock,sand/gravel/soil, sky/clouds, now/ice, trees/plants, vehicle, water, andmiscellaneous.
 10. The method of claim 1, where each of the one or morematerial classification scores represents the probability that a frameincludes content associated with each of the respective predefinedmaterial content categories.
 11. The method of claim 1, where the videofile comprises a shot of video.